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__ Complex Network Scalability  __ 


BGP is the protocol brains 
that controls the router brawn 
between different Internet 


service providers... 
33 


Boardwatch Magazine, April 1999, 
Scaling Internet and Data Services... 


_ Complex Network Scalability —_ 


Scalable : 
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° Scaling Your Network 
» Case Studies 


Troubleshooting 


° BGP Extensions 


Scaling Your Network 


Doing More with Less! 
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* Amount of routing information 
in the network 


Periodic updates/flooding 
Long convergence times 
Affects the core first 

° Policy definition 
Not easy to do 


_ BGP Cores—Sample Network __ 


* Geographically 
distributed 


° Hierarchical 
« Redundant 


° Media 
independent 


° Aclearly 
identifiable core 
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iBGP Core 
__ Migration Plan 


* Configure BGP in all the core routers 
Transit path 
Turn synchronization off 

» Route Generation 
Use static routes to create summaries 


Redistribution from the IGP is NOT 
recommended as it may cause instability 
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iBGP Core 
Migration Plan (Cont.) 


» Route Generation—Example: 
! 


router bgp 109 
network 200.200.200.0 
network 201.201.0.0 mask 255.255.0.0 


ip route 200.200.200.0 255.255.255.0 nullOd 
ip route 201.201.0.0 255.255.0.0 nullO 


317 
0901_04F9_c3 © , 


Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 
0901_04F9_c3.scr 


iBGP Core 
__ Migration Plan (Cont.) _ 


° Verify consistency of routing information 
Compare the routing table against 
the BGP table—they must match! 
» Change the distance parameters 
so that the BGP routes are preferred 
distance bgp 20 20 20 


All IGPs have a higher administrative distance 
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iBGP Core 
Migration Plan (Cont.) | 


° Filter “non-core” IGP routes 
Method will depend on the IGP used 


May require the use of a different IGP 
process in the core if using a link 
state protocol 


The routes to reach all the core links 
plus the BGP peering addresses 
must be carried by the IGP 
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iBGP Core Before... 


° IGP carries 
all the routes 


° The core routers 
may be stressed ‘ 
due to the large 
number of routes 
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° Core: 


IGP carries only 
core links plus 
peering address 
information 


BGP carries 
all the routes 


Increased Stability! 
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° The routes from the core cannot 
be redistributed back into the IGP 


Non-core areas need a default route 


Amount of routing information in 
non-core areas has been reduced! 


° Full logical iBGP mesh 


° External connections must be 
located in the core 
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ing Issues 


° Full mesh core 
High number of neighbors 
Update generation 
* Complex topologies 
Not a “simple” hierarchical network 


Multiple external and/or inter-region 
connections 


Policy definition and enforcement 
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Scaling Issues—Solutions 


* Reduce the number of updates 
Peer groups 


* Reduce the number of neighbors 
Confederations 
Route reflectors 


° Use additional information to 
effectively apply policies 
eBGP provides extra granularity 
Confederations 


Divide and Conquer! 


eBGP Connections and 
Confederations 
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Implementation Strategy —_ 


° Divide the network into multiple 
regions/areas 


* Connect each region using BGP 


* Reconfigure the IGP in each 
region/area 


_ Divide the Networkinto Pieces __ 


° Where: 
Geography 
Department lines | 
Hierarchy 
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eBGP Connections 
* Assign an ASN to each region 


Private ASNs maybe used and must be 
removed at the border of the network 


neighbor x.x.x.x remove-private-AS 
External connections only at the core 
° Apply policy at inter-AS borders 


May use AS _PATH filters to permit or 
deny route propagation to other regions 
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eBGP Connections (Cont.) 


» Only the routers connected 
to the core need to run BGP 


iBGP mesh in the core 


° ...Except if backdoor or transit 
connections exist 


Routers in the transit path need to 
run BGP too 
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AS65002 AS65004 


Transit AS65001 Backdoor 
Connection Connection 
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_ @BGP Connections—Routing —__ 


* Source the local routes for each AS at 
the border BGP routers 
Use static routes and network statements 
Verify consistency of routing information 


° What about the IGP? 


For each region/area it must carry routes 
to the infrastructure (all links), peering 
addresses and local destinations 

Filter at the borders 

May need to use an independent IGP 
process per AS 
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Confederations 


° Divide the AS into sub-AS 


eBGP between sub-AS, but some iBGP 
information is kept 


Preserve NEXT _HOP across the 
sub-AS (IGP carries this information) 


Preserve LOCAL PREF and MED 
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Confederations (Cont.) = 


* Visible to outside world as single AS 


Each sub-AS uses a number from the 
private space 


* iBGP speakers in sub-AS are 
fully meshed 
The total number of neighbors is reduced by 


limiting the full mesh requirement to only the 
peers in the sub-AS 


317 
0901_04F9_c3 © 1999, Cisco Systems, Inc. 


Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 
0901_04F9_c3.scr 


Confederations—NEXT_ HOP 


SUD-AS) —p 
65003 


Confederation 
100 
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Route Propagation Decisions 


« Same as with “normal” BGP: 


From peer in same sub-AS — only to external 
peers 


From external peers — to all neighbors 
° “External peers” refers to 
Peers outside the confederation 


Peers in a different sub-AS 
Preserve LOCAL PREF, MED and NEXT_HOP 
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Confederations—AS_ PATH 


* Sub-AS traversed are carried as part of 
AS_ PATH (AS_CONFED_SEQUENCE or 
AS _CONFED SET) for loop avoidance 


Not counted as regular AS when 
comparing AS_PATH 


Paths with only confederation ASNs 
in the AS_PATH are skipped during 
MED comparison 


bgp bestpath med confed 


© 1999, 


_Confederation—AS_PATH (Cont) _ 


== SUAS 
. 65001 


Confederation 
100 
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Confederations—Migration| _ 


» Same steps as when using eBGP 
connections, but external connections 
may be located anywhere in the network! 


° What about the IGP? 


It must carry routes to the infrastructure 
(all links) and peering addresses (including 
external NEXT_HOP) 


One instance of the IGP for the whole AS 


Confederations—Migration ll — 


° Migration from a full iBGP mesh may 
be tricky as all the routers must be 
configured at one time 


bgp confederation identifier real[ASN 
bgp confederation peers otherASNs 
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: ‘Migration 
Control Complexity 


Anywhere 
Confederations in the 
Network 


eBGP Low to 
Connections Instingss in) Medium 
Each) Region) 


Scalability and Stability Achieved by Both Methods! 
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Route Reflectors 


Playing with Mirrors 
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Route Reflectors 


° Provide additional control to allow 
router to advertise (reflect) iBGP 
learned routes to other iBGP peers 


Method to reduce the size of the iBGP mesh 
» Normal BGP speakers can coexist 


Only the RR has to support this feature 
neighbor x.x.x.x route-reflector-client 
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Clusters 
Clients Clients 


Lines Represent Both Physical Links and BGP Logical Connections 
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Route Reflectors— 
_ terminology (C 


Route reflector 
Router that reflects the iBGP information 


Client 


Routers between which the RR reflects updates (may be 
fully meshed among themselves) 


Cluster 


Set of one or more RRs and their clients 
(may overlap) 


Non-client 
iBGP neighbour outside the cluster 
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Route Reflectors— 
__ Loop Avoidance _ 


* Originator_ID attribute 


Carries the RID of the originator of the 
route in the local AS (created by the RR) 


° Cluster_list attribute 


The local cluster-id is added when the 
update is sent to (added by the RR) 


bgp cluster-id x.x.x.x 
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Reflection Decisions 


» Once the best path is selected: 
From non-client reflect to all clients 


From client > reflect to all non-clients 
AND other clients 


From eBGP peer -— reflect to all clients 
and non-clients 
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Route Reflectors—Hierarchy = __ 


* Clusters may be 
configured 
hierarchically 


RRs in a cluster are 
clients of RRs ina 
higher level 


Provides a 
“natural” 

method to limit 
routing information 
sent to lower levels 
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~ Hierarchical Route Reflectors | 


RR Se cluster-id 140.10.1.1 
Z x cluster-id 


RR 141.153.17.1 


x 


routerB>sh ip bgp 198.10.10.0 See 54 Pe 53.172 aay 
BGP routing table entry for 198.10.10.0/24 


3 
141.153.14.2 from 140.10.1.1 (141.153.17.2) 
Origin IGP, metric 0, localpref 100, valid, internal, best 


Originator : 141.153.17.2 


Cluster list: 144.10.1.1, 141.153.17.1 ane 


198.1009/ 


Lines represent both physical links 
and BGP logical connections 
317 
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RR Se cluster-id 140.10.1.1 
Z x cluster-id > 


RR 141.153.17.1 


~ 


routerB>sh ip bgp 198.10.10.0 as ereaa eas 
BGP routing table entry for 198.10.10.0/24 


3 
141.153.14.2 from 140.10.1.1 (141.153.17.2) 
Origin IGP, metric 0, localpref 100, valid, internal, best 


Originator : 141.153.17.2 
Cluster list: 144.10.1.1, 141.153.17.1 


Lines represent both physical links 
and BGP logical connections 
317 
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Hierarchical Route Reflectors _ 


RR Se cluster-id 140.10.1.1 
ea a cluster-id +) 


141.153.17.1 
pennants aneaane 141.153.30.1 Ss 
routerB>sh ip bgp 198.10.10. 141.153.17.2 


BGP routing table entry for 198.10.10.0/24 
3 


141.153.14.2 from 140.10.1.1 (141.153.17.2) 
Origin IGP, metric 0, localpref 100, valid, internal, best 


Originator : 141.153.17.2 
Cluster list: 144.10.1.1, 141.153.17.1 


Lines represent both physical links 
and BGP logical connections 


317 
0901_04F9_c3 © 1999, Cisco Systems, Inc. 


Route Reflectors—Redundancy __ 


» Multiple RRs can be configured 
in the same cluster 


Other RRs in the same cluster should 
be treated as iBGP peers (non-clients) 


All RRs in the cluster must have the 
same cluster-id 


* Arouter may be a client for RRs 
in different clusters 
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Multiple Route Reflectors 


cluster-id 1.1.1.1. 141.153.30.1 


RR | >a | RR 
—— — 
routerB>sh ip bgp 198.10.10.0 141.153.17.2 
BGP routing table entry for 198.10.10.0/24 


3 


141.153.14.2 from 141.153.30.1 (141.153.17.2) 
Origin IGP, metric 0, localpref 100, valid, internal, best 


Originator: 141.153.17.2 
Cluster list: 1.1.1.1 CS 9141.153.14.2 


— / 198.10.10.0/24s 


Lines Represent Both Physical 
Links and BGP Logical Connections 
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cluster-id 1.1.1.1 144153.30.1 


mi $a 
RR | >< | RR 
routerB>sh ip bgp 198.10.10.0 141.153.17.2 
BGP routing table entry for 198.10.10.0/24 


3 


141.153.14.2 from 141.153.30.1 (141.153.17.2) 
Origin IGP, metric 0, localpref 100, valid, internal, best 


Originator: 141.153.17.2 
Cluster list: 1.1.1.1 C=) 5 141.153.14.2 


— ) 198.10.10.0/24s 


Lines Represent Both Physical 
Links and BGP Logical Connections 
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Multiple Route Reflectors 


° The cluster-id 


must be different, Scar 
otherwise B Le NX Ve \ 
will not reflect <= p= 

any route to A “7 Ge Ge 
if coming from C 


: . Lines Represent Both Physical 
B will detect its own 


Links and BGP Logical Connections 


cluster-id in the cluster-list 


Tip: use a different cluster-id per RR 
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Route Reflectors—Migration =  _ 


° Where to place the route reflectors? 
Follow the physical topology! 


This will guarantee that the packet 
forwarding won’t be affected 


° Configure one RR ata time 
Eliminate redundant iBGP sessions 
Place one RR per cluster 
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Route Reflectors—Migration 


° Step 0: 
full iBGP 
mesh 


Logical Links 
Physical AND Logical Links 
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° Step 1: 
configure D 
as aRR;E 
is the client 


) 
ae Logical Links 
Physical AND Logical Links 
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Route Reflectors—Migration . 


° Step 2: 
eliminate 
unnecessary 
iBGP links 


Logical Links 
Physical AND Logical Links 


317 
0901_04F9_c3 © 1999, Cisco Systems, Inc. 


° Step 3: 
repeat for 
other clusters 
and iBGP 
links 


) 
eae Logical Links 
Physical AND Logical Links 
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RR: Other Issues 


° The set clause for outbound 
route-maps does not affect routes 
reflected to iBGP peers 


° The nexthop-self command will only 
affect the next-hop of eBGP learned 
routes (the next-hop of reflected 
routes should not be changed) 


Route Reflectors—Results 


* Number of neighbors is reduced 
No need for full iBGP mesh 


» Number of routes propagated is 
reduced 


Each RR advertises only the best path 
to its clients 


° Stability and Scalability are achieved! 
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Control 


Anywhere 
Confederations in the 


Network 


Route Anywhere 


Reflectors in the 
Network 
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Case Studies 


Common Problems and 
Troubleshooting 
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— Migration 


Complexity 


° RRs relax the logical 
full-mesh requirements 
that iBGP has 


Some configurations... 
“may not yield the same 
route-selection result as 
that of the full iBGP 
mesh...” 
draft-idr-route-reflect-v2, April 99 ee 


Connections 
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° Not following 
the physical 
topology 
may cause 
routing loops! 


Lines Represent 
Physical 
Connections 
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RR—Physical Topology . 


* Symptom 
routerC#traceroute 7.7.7.7 


Tracing the route to 7.7.7.7 
1 10.105.1.71 4 msec 4 msec 8 msec 
2 140.10.50.6 188 msec 4 msec 4 msec 
3 140.10.50.5 4 msec 4 msec 4 msec 
4 140.10.50.6 4 msec 8 msec 8 msec 
5 140.10.50.5 8 msec 8 msec 8 msec 
6 140.10.50.6 8 msec 4 msec 8 msec 
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routerA#show ip bgp 7.7.7.7 routerB#show ip bgp 7.7.7.7 
BGP routing table entry for 7.0.0.0/8 BGP routing table entry for 7.0.0.0/8 

1 1 

21.21.21.1 (metric 201) from 2.1.1.1 (2.1.1.1) 22.22.22.1 (metric 201) from 3.3.3.1 (3.3.3.1) 
Origin IGP,valid, internal, best Origin IGP, valid, internal, best 

routerA#show ip route 21.21.21.1 routerB#show ip route 22.22.22.1 
Routing entry for 21.21.21.0/24 Routing entry for 22.22.22.0/24 
Routing Descriptor Blocks: Routing Descriptor Blocks: 

* 140.10.50.6, from 140.10.50.6, via Serial * 140.10.50.5, from 140.10.50.5, via Serial 
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RR—Physical Topology 


° Solution: 
Follow the 
physical topology! 


Lines Represent 
Physical 
Connections 
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* Symptom 
routerD#traceroute 7.1.1.1 
1 1.1.1.2 24 msec 24 msec 40 msec 
rirB 2 156.1.1.1 28 msec 48 msec 24 msec 
rwC 3 156.1.1.2 24 msec 24 msec 24 msec 
4 156.1.1.1 28 msec 28 msec 24 msec 
5 156.1.1.2 28 msec 28 msec 28 msec 
6 156.1.1.1 28 msec 28 msec 32 msec 


Lines Represent 
Physical Connections 
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RR—Physical Topology Il 


routerC#show ip bgp 7.0.0.0 routerB#show ip bgp 7.0.0.0 
BGP routing table entry for 7.0.0.0/8 BGP routing table entry for 7.0.0.0/8 
1 1 
150.10.10.1 (metric 115) from 150.10.10.1 (150.20.20.1) 156.1.1.2 from 156.1.1.2 (212.212.212.1) 
Origin IGP, valid, external, best Origin IGP, valid, internal, best 

routerC#show ip route 150.10.10.1 routerB#show ip route 156.1.1.2 
Routing entry for 150.10.10.1/32 Routing entry for 156.1.1.0/24 
Routing Descriptor Blocks: Routing Descriptor Blocks: 

* 156.1.1.1, from 150.20.20.1, via Ethernet2/1/1 * directly connected, via Ethernet1 
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RR—Physical Topology Ill 


° Problem 


routerC#show running-config 

router bgp 134 

neighbor 150.10.10.1 remote-as 1 

neighbor 150.10.10.1 ebgp-multihop 255 
neighbor 150.10.10.1 update-source LoopbackO 
neighbor 156.1.1.1 remote-as 134 

neighbor 156.1.1.1 route-reflector-client 
neighbor 156.1.1.1 next-hop-self 

! 


Lines Represent 
Physical Connections 
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° Problem 


routerC#show running-config 
router bgp 134 

neighbor 150.10.10.1 remote-as 1 

neighbor 150.10.10.1 ebgp-multihop 255 

neighbor 150.10.10.1 update-source LoopbackO 

neighbor 156.1.1.1 remote-as 134 

neighbor 156.1.1.1 route-reflector-client 

neighbor 156.1.1.1 next-hop-self 

! 

ip route 150.10.10.1 255.255.255.255 sO 250 Lines Represent 


Physical Connections 
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° Solution 


Establish the eBGP 
peering permanently 
through the 
“backup” link 


Use LOCAL_PREF or 
MED to break any tie! 


Lines Represent 
Physical Connections 
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Clusters with MultipleRRs =, 


° It is possible to 
have multiple RRs 
in one cluster for 
redundancy 


° Hierarchical 
clusters help ea 
scale your network 
Lines Represent Physical 


and Logical Connections 
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Clusters with MultipleRRs _ 


- AandB are 
core routers 


Carry routes to the 
rest of the network 


Cluster-id 5 
° Symptom Lf S 


RR-C is not 
receiving any routes 


Lines Represent Physical 
and Logical Connections 
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Clusters with Multiple RRs 


° Problem 


After resetting the session and using 
debug ip bgp: 
BGP: 1.1.1.1 Route Reflector cluster loop received cluster-id 0.0.0.5 
BGP: 2.2.2.2 Route Reflector cluster loop received cluster-id 0.0.0.5 
C is configured with the same cluster-id 
as A and B! 


routerC: 

! 

router bgp 1 
bgp cluster-id 5 
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Clusters with MultipleRRs _ 


° Solution 


In hierarchical route reflector 
configurations, each level must 
have a different cluster-id 


Recommendation: use a different 
cluster-id per route reflector 
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* Symptom 


The eBGP peering is established, but 
convergence is not complete even after 
several hours 


routerA#show ip bgp summary 


Neighbor V AS MsgRcvd MsgSent TbliVer InQ OutQ Up/Down State/PfxRed 
150.10.10.1 4 1 3550 3570 847 0 206 05:53:51 100 
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eBGP Multihop 


Reply to request 0 
Record route: 
(156.1.1.2) 
routerA#show ip route 150.10.10.1 (195.5.5.1) 


Routing entry for 150.10.10.1/32 ti 30.10.10.1)" 


; ; : (10.105.1.76) 
Routing Descriptor Blocks: (195.5.5.2) 


10.105.1.71, from 150.20.20.1, 00:06:14 ago, via POS2/1/0 (196.1.1-1) 
* 156.1.1.1, from 150.20.20.1, 00:06:14 ago, via POS2/1/1 iter ely 


routerA#ping 150.10.10.1 peply to renueee 


Sending 5, 100-byte ICMP Echos to 150.10.10.1: — !!I! {i ae 10505) 

Success is 100 percent, round-trip min/avg/max = 4/64/296 ms (1 50.10.10.1) 
(140.10.50.6) 
(10.105.1.71) 
(211.211.211.1) <*> 
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eBGP Multihop —— 


° Problem: peers configured 
with eBGP-multihop 2 


eBGP Peering 
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eBGP Multihop 


° Solution 


The paths have different number of hops 
between them—make sure that the TTL 
is enough for the longest path 
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Common Problems—Conclusions 


° BGP is a simple protocol 
Straight forward state machine 
Rides over TCP 
Easy “basic” configuration 

° BGP is also very flexible 


Many options and knobs! 


BGP Extensions 


There’s More! 
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OPEN Message 


0123 4 5 6 7 8 910 11 12 13 14 15 16 17 18 1920 21 22 23 24 25 26 27 28 29 30 31 


Version 
My Autonomous System 
Hold Time 


BGP Identifier 


Opt. Parm: Len. 


Optional Parameters 
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Capabilities Negotiation 


° Allows for the 
advertisement of 


capabilities (type 2) Capability Code (1 Octet) 


- Backwar mpatibl 
ackwards compatible Capability Length (1 Octet) 
New error subcode 


introduced to indicate Capability Value (Variable) 


which capabilities are 
not supported—the 
session must be reset draft-ietf-idr-bgp4-cap-neg, Feb. 1999 
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Capabilities Negotiation . 


* Current capabilities 
1 multiprotocol 
128 route refresh 
129 outbound route filter 
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Route Refresh Capability 


° Facilitates non-disruptive 
policy changes 


* No configuration is needed 
» No additional memory is used 
¢ Clear ip bgp x.x.x.x [soft] in 
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Managing Policy Changes 

clear ip bgp <addrs [soft] [in|out] 

» <addr> may be any of the following 
X.X.X.X IP address of a peer 
7 all peers 
ASN all peers in an AS 
external all external peers 
peer-group <name> all peers in a peer-group 
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Outbound Route Filter Capability _ 


* Allows for the use of the neighbor’s 
inbound prefix-list as part of the local 
outbound policy (Currently only for 
IPv4 unicast NLRI) 


Reduces the number of updates 


5 sec. delay after session is established, 
before updates are sent 
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PrefixList-ORF 


* By default, this capability is not 
advertised to any neighbor 


neighbor x.x.x.x capability prefix-filter 


Can’t be advertised to peer-group 
members 


° To push out a prefix-list 
clear ip bgp x.x.x.x in prefix-list 


Also requests a route refresh 
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Multiprotocol Extensions— 
_tic2283 


| Address Family Identifier (2 Octets) 
| | Subsequent Address Family Identifier (1 Octet) 


a Length of Next Hop Network Address (1 Octet) 


ad Network Address of Next Hop (Variable) — 


Number o of First ‘SNP As (1 Octet) — 


2 Length of First SNP A(1 Octet) = = 


Length of First SNP A (4 Octet) a 
| First SNP A (Variable) a 


| Length of Last SNPA (1 Octet) _ 


| Le Last SNP A (Variable) 


_ Network layer Reachability ity Information (Variable) = 
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~ Address Family Identifiers . 


* Address family identifier—rfc1700 
1 IPv4 
2 IPv6 
8 E.164 


> Sub-AFI (for IPv4) 
1 unicast 
2 multicast 
3 unicast + multicast 
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_ Multiprotocol Extensions! __ 


° mBGP 


Used to propagate multicast source 
information 


° The different NLRI types allow for 
diverging topologies 


The NEXT_HOP information is different 
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_____ MultiprotocolExtensionsit[ 


*» MPLS VPN 


Used to carry both intra- and 
inter-VPN routing information 


° New AFI—VPN-IPv4 


° NLRI format for VPN addresses 
Tag 

VPNID (32 bits) 

Prefix (variable length, 0-32 bits) 
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ded Community Attribute —__ 


* Extended range 
8 octets 
» Structure 
Type: value 
Value may be of the form AS:xxx 


» Same functionality as existing attribute 


draft-ramachandra-bgp-ext-communities, March 1999 
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_ Complex Network Scalability . 


° Scalable 


Confederations, route reflectors, and 
multiprotocol support 


° Stable 


Network isolation, capability to 
handle large amount of data 


° Simple 
... But flexible and extendible 
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For Further Reference: 


* Advanced IP Network Design 
White, et. All—Cisco Press 1999 


° BGP4 
Stewart—Addison Wesley 1999 


¢ Internet Routing Architectures 
Halabi—Cisco Press 1997 


¢ IETF IDR Working Group 
(http ://www.ietf.org) 
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